System and method for communicating the synchronization status of memory modules during initialization of the memory modules

ABSTRACT

A memory system includes a memory hub controller coupled to a plurality of memory modules each of which includes a memory hub. The memory hub controller and the memory hubs each include at least one receiver that is synchronized to an internal clock signal during initialization. The memory hub controller and the memory hubs each transmit an initialization complete signal downstream when at least one receiver in the controller or hub is initialized and, in the case of the memory hubs, when a downstream initialization signal has also been received. Similarly, the memory hubs transmit an initialization signal upstream to another memory hub or the controller when both of its receivers are initialized and an upstream initialization signal has also been received. Receipt of an upstream initialization signal by the memory hub controller signifies that all of the receivers have been initialized.

TECHNICAL FIELD

The present invention relates to a processor-based system, and moreparticularly, to a processor-based system having a memory module with amemory hub coupling several memory devices to a processor or othermemory access device.

BACKGROUND OF THE INVENTION

Processor-based systems, such as computer systems, use memory devices,such as dynamic random access memory (“DRAM”) devices, as system memoryto store instructions and data that are accessed by a processor. In atypical computer system, the processor communicates with the systemmemory through a processor bus and a memory controller. The processorissues a memory request, which includes a memory command, such as a readcommand, and an address designating the location from which data orinstructions are to be read or to which data or instructions are to bewritten. The memory controller uses the command and address to generateappropriate command signals as well as row and column addresses, whichare applied to the system memory. In response to the commands andaddresses, data is transferred between the system memory and theprocessor. The memory controller is often part of a system controller,which also includes bus bridge circuitry for coupling the processor busto an expansion bus, such as a PCI bus.

Although the operating speed of memory devices has continuouslyincreased, this increase in operating speed has not kept pace withincreases in the operating speed of processors. Even slower has been theincrease in operating speed of memory controllers coupling processors tomemory devices. The relatively slow speed of memory controllers andmemory devices limits the data bandwidth between the processor and thememory devices.

One approach to increasing the data bandwidth to and from memory devicesis to use multiple memory devices coupled to the processor through amemory hub as shown in FIG. 1. A computer system 10 using a memory hubarchitecture includes a processor 104 for performing various computingfunctions, such as executing specific software to perform specificcalculations or tasks. The processor 104 includes a processor bus 106that normally includes an address bus, a control bus, and a data bus.The processor bus 106 is typically coupled to cache memory 108, which,is typically static random access memory (“SRAM”). Finally, theprocessor bus 106 is coupled to a system controller 110, which is alsosometimes referred to as a bus bridge.

The system controller 110 contains a memory hub controller 112 that iscoupled to the processor 104. The memory hub controller 112 is alsocoupled to several memory modules 114 a–n through a bus system 115. Eachof the memory modules 114 a–n includes a memory hub 116 coupled toseveral memory devices 118 through command, address and data buses 117.The memory hub 116 efficiently routes memory requests and responsesbetween the controller 112 and the memory devices 118. Computer systemsemploying this architecture can have a higher bandwidth because theprocessor 104 can access one memory module 114 a–n while another memorymodule 114 a–n is responding to a prior memory access. For example, theprocessor 104 can output write data to one of the memory modules 114 a–nin the system while another memory module 114 a–n in the system ispreparing to provide read data to the processor 104. The operatingefficiency of computer systems using a memory hub architecture can makeit more practical to vastly increase data bandwidth of a memory system.A memory hub architecture can also provide greatly increased memorycapacity in computer systems.

The system controller 110 also serves as a communications path to theprocessor 104 for a variety of other components. More specifically, thesystem controller 110 includes a graphics port that is typically coupledto a graphics controller 121, which is, in turn, coupled to a videoterminal 123. The system controller 110 is also coupled to one or moreinput devices 120, such as a keyboard or a mouse, to allow an operatorto interface with the computer system 10. Typically, the computer system10 also includes one or more output devices 122, such as a printer,coupled to the processor 104 through the system controller 110. One ormore data storage devices 122 are also typically coupled to theprocessor 104 through the system controller 110 to allow the processor104 to store data or retrieve data from internal or external storagemedia (not shown). Examples of typical storage devices 124 include hardand floppy disks, tape cassettes, and compact disk read-only memories(CD-ROMs).

Although there are advantages to utilizing a memory hub for accessingmemory devices, the design of the hub memory system, and more generally,computer systems including such a memory hub architecture, becomesincreasingly difficult. The memory modules 114 a n each internallyoperate synchronously although operations between memory modules areasynchronous. The command, address, and data signals transferred to thememory module 114 a n are normally latched or strobed into the memorymodules 114 a n by a clock signal. However, as transfer rates increase,the time during which the command, address and data signals as receivedat the memory hubs 116 are valid decreases. This period during which thesignals are valid is commonly referenced by those ordinarily skilled inthe art as the “window” or “eye.” Not only does the size of the eye forcommand, address, and data signals decrease, but the time or location ofthe eye can also vary because of various factors, such as timing skew,voltage and current drive capability, and the like. In the case oftiming skew of signals, it often arises from a variety of timing errorssuch as loading on the lines of the bus and the physical lengths of suchlines.

As the size of signal eyes decrease at higher transfer rates, thevariations in the location of the signal eyes become more of a problem.One technique to alleviate this problem to some extent is to couple aclock to the memory modules. As shown in FIG. 1, a clock generator 500generates a clock signal CLK and couples it to the memory hub controller112 and each of the memory hubs 116 in respective memory modules 114 a–nalong with the command, address and data signals that are coupledthrough the bus system 115. The CLK signal is coupled from the memoryhub controller 112 in synchronism with the command, address and datasignals so it, in theory, should be usable by the memory hubs 116 todefine the eye during for the command, address and data signals as theyare received at the memory hubs 116. However, in practice, even thisapproach becomes ineffective as signal transfer rates continue todecrease. In particular, the CLK signal may be subject to differentconditions than the command, address and data signals, such as beingcoupled through a physically different signal path or being loaded to agreater degree. Also, for the clock forwarding techniques used in thecomputer system 10 to successfully function at higher clock speeds, thelayout of conductors between the memory hub controller 112 and thememory hubs 116 must be precisely controlled.

One technique that has been proposed to allow the CLK signal to continuebeing used to strobe command, address and data signals at highertransfer rates is to include circuitry (not shown) in the memory hubs116 that adjusts the timing of the CLK signal within each of the hubs116 so that it is aligned with the signal eye. However, this techniqueadds a fair degree of complexity to the memory hubs 116 and is notalways effective.

There is therefore a need for a system and method that allows command,address and data signals to be coupled between a memory hub controllerand one or more memory hubs in respective memory modules that avoidsproblems of synchronizing a clock signal coupled between the memory hubcontroller and memory hubs along with the command, address, and datasignals.

SUMMARY OF THE INVENTION

A memory system includes a memory hub controller coupled to a pluralityof memory hubs each of which includes at least one receiver that isinitialized before use. The memory hub controller transmits a downstreaminitialization complete signal to one of the memory hubs wheninitialization of a receiver in the memory hub controller has beencompleted. Each of the memory hubs determines if it is receiving arespective downstream initialization complete signal, and it alsodetermines if initialization of at least one internal receiver has beencompleted. If so, each of the memory hubs except for a final memory hubfurthest downstream from the memory hub controller transmits adownstream initialization complete signal. If the final memory hubreceives the respective downstream initialization complete signal andinitialization of at least one receiver in the final memory hub has beencompleted, the final memory hub transmits an upstream initializationcomplete signal. Each of the memory hubs except for the final memory hubdetermines if it is receiving a respective upstream initializationcomplete signal, and it also determines if initialization of allreceivers in the memory hub has been completed. If so, the memory hubtransmits an upstream initialization complete signal. When the memoryhub controller receives a respective upstream initialization completesignal, it transitions the memory system to normal operation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer system that includes severalmemory modules having a memory hub architecture coupled to a memory hubcontroller.

FIG. 2 is a block diagram of a computer system that includes severalmemory modules having a memory hub architecture according to oneembodiment of the present invention.

FIG. 3 is a block diagram of one embodiment of receivers andtransmitters used in the computer system of FIG. 2 or some other systemin accordance with the present invention.

FIG. 4 is a flow chart showing one embodiment of a technique forcommunicating the synchronization status of each memory module used inthe computer system of FIG. 2 during initialization so that a memory hubcontroller used in the system of FIG. 2 can determine wheninitialization of all memory modules have been completed.

FIG. 5 is a block diagram of a memory hub that may be used the memorymodules that are used in the computer system of FIG. 2.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention are directed to a memory module andmemory hub controller each having the capability of generating a clocksignal for strobing data signals during the “eye” of the data signalswhen the data signals are valid. More particularly, embodiments of thepresent invention are directed to a system and method for communicatingthe initialization status of the memory module and memory hubcontroller. Certain details are set forth below to provide a sufficientunderstanding of various embodiments of the invention. However, it willbe clear to one skilled in the art that the invention may be practicedwithout these particular details. In other instances, well-knowncircuits, control signals, and timing protocols have not been shown indetail in order to avoid unnecessarily obscuring the invention.

A computer system 100 having a hub memory system according to oneembodiment of the invention is shown in FIG. 2. The computer system 100uses many of the same components that are used in the computer system 10of FIG. 1. Therefore, in the interest of brevity, these components havebeen provided with the same reference numerals, and an explanation oftheir the functions and operation will not be repeated.

As in the computer system 10 of FIG. 1, the system controller 110 alsoincludes a memory hub controller 128 that is coupled to several memorymodules 130 a,b . . . n, which serve as system memory for the computersystem 100. The memory modules 130 are each coupled to a high-speeddownstream bus 132 and a high-speed upstream bus 134. The downstream bus132 extends downstream from the memory hub controller 128 and bothupstream and downstream from each of the memory modules 130 except thememory module 130n furthest from the memory hub controller 128.Similarly, the upstream bus 134 extends upstream from each of the memorymodules 130 and downstream from all of the memory modules 130 except thememory module 130 n. Each of these buses 132, 134, include a discretedata bus, although they may also include discrete command and addressbuses, a combined command/address bus, or some other bus system.However, the explanation of the various embodiments will be with respectto a data bus, it being understood that a similar technique can be usedto strobe command and address signals.

The downstream buses 132 couple data away from the memory hub controller128, and the upstream buses 134 couple data toward the memory hubcontroller 128. Therefore, the downstream bus 132 couples write data toand from each of the memory modules 130, except for the memory module130n furthest downstream from the memory hub controller 128, which onlyreceives write data. Similarly, the upstream bus 134 couples read datato and from each of the memory modules 130, except for the memory module130n furthest downstream from the memory hub controller 128, which onlytransmits read data. The downstream bus 132 also couples write data fromthe memory hub controller 128, and the upstream bus 134 couples readdata to the memory hub controller 128. Significantly, the buses 132, 134need not couple clock signals to and from the memory modules 130 and thememory hub controller 128 for the purpose of allowing the memory modules130 to capture data transmitted through the buses 132, 134. Instead, asexplained in greater detail below, each of the memory modules 130 andthe memory hub controller 128 generates signals internally to strobe thedata coupled through the buses 132, 134.

The memory modules 130 are shown coupled to the memory hub controller128 in a point-to-point coupling arrangement in which each of the buses132 and 134 are coupled only between two points. However, it will beunderstood that other topologies may also be used. For example, it maybe possible to use a multi-drop arrangement in which a single downstreambus (not shown) and a single upstream bus (not shown) are coupled to allof the memory modules 130. A switching topology may also be used inwhich the memory hub controller 128 is selectively coupled to each ofthe memory modules 130 through a switch (not shown). Other topologiesthat may be used will be apparent to one skilled in the art.

Each of the memory modules 130 includes a first receiver 142 thatreceives write data through the downstream bus 132, a first transmitter144 that transmits read data upstream through the upstream bus 134, asecond transmitter 146 that transmits write data downstream through thedownstream bus 132, and a second receiver 148 that receives read datathrough the upstream bus 134.

The memory modules 130 also each include a memory hub local 150 that iscoupled to its first receiver 142 and its first transmitter 144. Thememory hub local 150 receives write data through the downstream bus 132and the first receiver 142 and couples the write data to one or more ofsixteen memory devices 160, which, in the example illustrated in FIG. 2,are synchronous dynamic random access memory (“SDRAM”) devices. However,a fewer or greater number of memory devices 160 may be used, and memorydevices other than SDRAM devices may also be used. The memory hub local150 is coupled to each of the memory devices 160 through a bus system164, which normally includes a control bus, an address bus, and a databus. However, other bus systems, such as a bus system using a sharedcommand/address bus, may also be used.

The memory hub 150 also receives read data from one or more of thememory devices 160 and couples the read data through the firsttransmitter 144 and the upstream bus 134. In the event the write datacoupled through the downstream bus 132 and the first receiver 142 is notbeing directed to the memory devices 160 in the memory module 130receiving the write data, the write data are coupled though a downstreambypass path 170 to the second transmitter 146 for coupling through thedownstream bus 132. Similarly, if read data is being transmitted from adownstream memory module 130, the read data is coupled through theupstream bus 134 and the second receiver 148. The read data are thencoupled upstream through an upstream bypass path 174, and then throughthe first transmitter 144 and the upstream bus 134. The second receiver148 and the second transmitter 146 in the memory module 130 n furthestdownstream from the memory hub controller 128 are not used and may beomitted from the memory module 130 n.

The memory hub controller 128 also includes a transmitter 180 coupled tothe downstream bus 132, and a receiver 182 coupled to the upstream bus134. The downstream bus 132 from the transmitter 180 and the upstreambus 134 to the receiver 182 are coupled only to the memory module 130 athat is the furthest upstream to the memory hub controller 128. Thetransmitter 180 couples write data from the memory hub controller 128,and the receiver 182 couples read data to the memory hub controller 128.

The computer system 100 also includes a reference clock generator 190,which generates a reference clock signal that is coupled to the memoryhub controller 128 and each of the memory modules 130. The memory hubcontroller 128 and the memory modules 130 use the reference clock togenerate an internal clock signal that, in the embodiment of FIG. 2, hasa frequency of four times the frequency of the reference clock signal.This internal clock signal is used as a transmit clock to strobe datafrom the transmitters 144, 146, 180. The receivers 142, 148, 182 adjustthe phase of the internal clock signal to generate a receive clocksignal that is used to strobe data into the receivers 142, 148, 182.Briefly, the receivers 142, 148, 182 perform this function by receivinga known data pattern from a transmitter 144, 146, 180 to which it iscoupled, and attempt to capture that data pattern by strobing the dataas the phases of the receive clock signals are incrementally varied. Thephase of the receive clock signal that best captures the data pattern isthen used to strobe data into the receivers 142, 148, 182 in normaloperation.

One embodiment of the receivers 142, 182 and the transmitters 144, 180in the memory hub controller 128 and in one of the memory modules 130 isshown in FIG. 3. In both cases, a receiver 200 functions as bothreceivers 142, 148 in the memory module 130 and the receiver 182 in thememory hub controller 128, and a transmitter 210 functions as bothtransmitters 144, 146 in the memory module 130 as well as the singletransmitter 180 in the memory hub controller 128. The transmitter 210includes a pattern generator 220 that generates a predetermined patternof data bits, and a transmit interface control 224 that controls thetransmitting of the pattern. In the embodiment of FIG. 3, the samepattern is transmitted on all of the data bits of the buses 132, 134.During an initialization phase of the receiver 200 and transmitter 210,the transmitter 210 continuously repeats its transmission of the datapattern.

As previously explained, the receiver 200 receives the data bits fromthe transmitter 210 and strobes them in using a receive clock signalgenerated from the clock signal received from the clock generator 500and having four times the frequency of the core clock. Morespecifically, in one embodiment of the invention, the patterntransmitted by the transmitter 210 is the following 32-bit patterndivided into four cycles each having 8 bits: “01011011 11000101 1001001100101100” (hex “5BC5932C”). The data bit pattern is transmitted fromright to left. In the embodiment of FIG. 3, a bit is strobed into thereceiver 200 on each transition of the receive clock signal, so two bitsare captured by the receiver 200 on each receive clock cycle. Since thereceive clock has a frequency of four times the core clock, eight bitsof data are captured during each cycle of the core clock.

In the embodiment of FIG. 3, the first bit is always captured on thepositive edge of the receive clock signal. As a result, there are 16possible patterns of valid data captured by the receiver 200, namely,the transmitted 32-bit pattern shifted by two bits for each pattern. Anexpected pattern memory 230 stores all 16 of these possible patterns,which, as previously explained, consists of eight bits.

It is significant, for reasons that will be explained below, that thetransmitters 210 can transmit the inverse of the above 32-bit pattern,namely “10100100 00111010 01101100 11010011, ” and the receivers 200 candetect that pattern as well as the fact the inverse of the pattern isbeing transmitted.

In the embodiment of FIG. 3, a pattern comparator 234 performs threecomparisons. First, it checks all of the data bits of the bus 132 toensure that they all have the same value as each data bit is capturedsince the same data are transmitted on each data bit of the bus 132.This same comparison is performed on bus 134.

In the second comparison, the pattern comparator 234 compares the eightdata bits captured in the receiver 200 for each core cycle to thesixteen valid 8-bit data bit patterns stored in an expected patternmemory 230. For purposes of this comparison, it can use any of the 32bits captured on each transition of the receive clock signal since thefirst comparison confirmed that all 32 bits were the same. Based on thiscomparison, phase adjustment logic 240 adjusts the phase of the receiveclock signal so that it can best capture the data coupled to thereceiver. More specifically, the pattern comparator 234 compares the 8bits received during any core cycle to the 16 valid patterns and theirinverses stored in the expected pattern memory 230 to adjust the phaseof the receive clock signal. The above operation is controlled by areceive interface controller 244. One embodiment of the receivers 200 isshown and described in greater detail, and the operation of thetransmitters 210 and receivers are described in greater detail, in U.S.patent application, Ser. No. * having a common inventor, which isincorporated herein by reference.

In the third comparison, the pattern comparator 234 checks an additional33^(rd) bit, which functions as a control bit. The pattern that is senton the beses 132, 134 is also sent on the control bit for each of thesebuses. The eight bits captured on one core clock is compared in the samemanner as the second comparison.

The duration of the initialization process during which the phase of thereceive clock signal is adjusted will typically be different for each ofthe receivers 200 used in the memory modules 130. It is important forthe receivers 200 to be able to provide an indication of when theadjustment of the phase of the receive clock signal has been completedfor a least two reasons. First, once initialization of a receiver 200has been completed, the transmitter 210 no longer needs to transmit the32-bit initialization patterns. By communicating its initializationstatus, the receiver 200 can inform the transmitter 210 when it candiscontinue transmitting the 32-bit pattern.

The second reason why it is important for the receivers 200 to be ableto communicate their respective synchronization status is because thecomputer system 100 (FIG. 2) cannot begin normal operation untilinitialization of all of the receivers 200 in the memory modules 130have been completed. Once all of the memory modules 130 havecommunicated their initialization completion to the memory hubcontroller 112 (FIG. 2), the memory hub controller 112, and the entirecomputer system 100, can begin normal operation.

One embodiment of a technique by which the receivers 200 can communicatetheir respective initialization status will now be explained withreference to the flow chart of FIG. 4. The initialization process isentered at step 300, and the transmitter 180 (FIG. 2) in the memory hubcontroller 128 begins transmitting test data, i.e., the 32-bit pattern“01011011 11000101 10010011 00101100, ” at step 304. Each memory module130 checks at step 310 to determine if its receiver 142 that is coupledto the downstream bus 132 is receiving test data. When the receiver 142in a memory module begins receiving test data, the transmitter 144coupled to the upstream bus 134 begins transmitting test data at step314. Also, the detection of test data on the downstream bus 132 at step310 causes the transmitter 146 coupled to each downstream bus 132 tobegin transmitting test data at step 318. Thus, in a system 100 withmultiple memory modules, the transmission of test data by the memory hubcontroller 128 at step 304 causes each downstream memory module 130 tosuccessively transmit test data both upstream and downstream until bothof the transmitters 144, 146 in all of the memory modules 130 aretransmitting test data. The receiver 182 in the memory hub controller128 and the receivers 142, 148 in each of the memory modules 130 thenuse the test data to adjust the phase of their respective internallygenerated receive clock signals, as previously explained.

The procedure shown in FIG. 4 then checks at step 320 to determine whenadjustment of the phases of the receive clock signals generated by thereceiver 182 in the memory hub controller 128 or the receivers 142, 148in each memory module 130 has been completed. When phase adjustmentcompletion is detected at step 320, the transmitter 180 in the memoryhub controller 128 or the “downstream” transmitter 146 in each memorymodule 130 checks at step 324 to determine if the original test data(i.e., non-inverted test data) are still being received through thedownstream bus 132. If a determination is made at step 324 that theoriginal test data is not being received, the “downstream” transmitter180 or 132 in the memory hub controller 128 or memory module 130,respectively, inverts the test data it transmits on the downstream bus132 at step 328.

There is no receiver coupled to the downstream bus 132 for the memoryhub controller 128 so the determination at step 324 will always resultin the transmitter 180 inverting the test data at step 328 as soon asinitialization of the receiver 182 has been completed. However, the“downstream” transmitters 146 in the respective memory modules 130 willnot couple inverted test data to the downstream bus 132 at step 328until detection of initialization completion of both receivers 142, 148is detected at step 320 and receipt of inverted test data on thedownstream bus 132 is detected at step 324. As a result, when the memoryhub controller 128 or each of the memory modules 130 transmits invertedtest data on the downstream bus 132, it notifies the downstream memorymodule that initialization has been completed. Each memory module 130thus transmits inverted data downstream in sequence upon receipt ofinverted test data from either the adjacent memory hub controller 128 orthe upstream adjacent memory module 130 as long as initialization ofboth of its receivers have been completed. When the furthest downstreammemory module 130 receives inverted test data on the downstream bus 132at step 324, all of the memory modules 130 have been notified thatinitialization has been completed for all upstream memory modules andthe memory hub controller 128.

Normal operation cannot begin responsive to the furthest downstreammemory module 130 receiving inverted test data on the downstream bus 132because the memory hub controller 128 and the memory modules 130 havenot been notified that initialization has been completed for all of thedownstream memory modules 130. Therefore, the process checks at step 330to determine if inverted test data have been received through anyupstream data bus 134. The memory module 130 furthest downstream fromthe memory hub controller 128 has no upstream data bus 134, so the“upstream” transmitter 144 in the furthest downstream memory modulebegins transmitting inverted test data upstream to the upstream memorymodule 130 at step 336. In like manner, each “upstream” receiver 148checks at step 330 to determine if it is receiving inverted test data.If so, the upstream transmitter 144 in the memory module 130 likewisebegins transmitting inverted test data upstream at step 336. When thereceiver 182 in the memory hub controller 128 detects inverted test dataat step 340, it is notified that initialization in both receivers 142,148 in all of the memory modules 130 have been completed. The memory hubcontroller 128 then switches to the normal operating mode at step 344 bysending NOP's, or “No Operations” on the downstream bus 132.

In an alternative embodiment, the completion of initialization for onlythe downstream receiver 142 is checked at step 320 so that thedownstream transmitter 146 in each memory module 130 will begintransmitting inverted test data at step 328 as soon as inverted testdata is detected on the downstream bus 132 at step 324. In thisembodiment, when the furthest downstream memory module detects invertedtest data, it only signifies that initialization of the downstreamreceivers 142 have been completed. However, in this embodiment, theinitialization status of the downstream receivers 148 are checked atstep 320 along with the determination of whether inverted test data isbeing received by the “upstream” receiver 148 in the memory module. Ifinverted test data is being received through the upstream receiver 148and initialization of the upstream receiver has been completed, theupstream transmitter 144 begins transmitting inverted test data at step336. Therefore, receipt of inverted test data by the receiver 182 in thememory hub controller 128 still signifies that initialization for bothreceivers 142, 148 in all of the memory hubs 130 have been completed.

Although the transmission and detected of inverted test data providesone means of communicating the initialization status of the memory hubcontroller 128 and the memory modules, other communication protocols mayalternatively be used.

FIG. 5 shows an embodiment of the memory hub local 150 according to thepresent invention, which can be used in the memory modules 130 of FIG.2. The memory hub local 150 includes two input bus interfaces 410 a,d,which may be used to couple data into the memory hub local 150, and twooutput bus interfaces 412 a,b, which may be used to couple data from thememory hub the memory hub local 150.

The bus interfaces 410 a,b, 412 a,b are coupled to a switch 460 througha plurality of bus and signal lines, represented by buses 414. The buses414 are conventional, and include a write data bus and a read data bus,although a single bi-directional data bus may alternatively be providedto couple data in both directions through the bus interfaces 410 a,b,412 a,b. It will be appreciated by those ordinarily skilled in the artthat the buses 414 are provided by way of example, and that the buses414 may include fewer or greater signal lines, such as further includinga request line and a snoop line, which can be used for maintaining cachecoherency.

The switch 460 is coupled to four memory interfaces 470 a–d which are,in turn, coupled to the memory devices 160 (FIG. 2). By providing aseparate and independent memory interface 470 a–d for each set of memorydevices 160, the memory hub local 150 avoids bus or memory bankconflicts that typically occur with single channel memory architectures.The switch 460 is coupled to each memory interface through a pluralityof bus and signal lines, represented by buses 474. The buses 474 includea write data bus, a read data bus, and a request line. However, it willbe understood that a single bi-directional data bus may alternatively beused instead of a separate write data bus and read data bus. Moreover,the buses 474 can include a greater or lesser number of signal linesthan those previously described.

In an embodiment of the present invention, each memory interface 470 a–dis specially adapted to the memory devices 160 to which it is coupled.More specifically, each memory interface 470 a–d is specially adapted toprovide and receive the specific signals received and generated,respectively, by the memory devices 160 to which it is coupled. Also,the memory interfaces 470 a–d are capable of operating with memorydevices 160 operating at different clock frequencies. As a result, thememory interfaces 470 a–d isolate the processor 104 from changes thatmay occur at the interface between the memory hub 130 and memory devices160 coupled to the memory hub local 150, and it provides a morecontrolled environment to which the memory devices 160 may interface.

The switch 460 coupling the bus interfaces 410 a,b, 412 a,b and thememory interfaces 470 a–d can be any of a variety of conventional orhereinafter developed switches. For example, the switch 460 may be across-bar switch that can simultaneously couple bus interfaces 410 a,b,412 a,b to each other to provide the downstream bypass path 170 and theupstream bypass path 174 shown in FIG. 2. The switch 460 can also be aset of multiplexers that do not provide the same level of connectivityas a cross-bar switch but nevertheless can couple the some or all of thebus interfaces 410 a,b, 412 a,b to each of the memory interfaces 470a–d. The switch 460 may also includes arbitration logic (not shown) todetermine which memory accesses should receive priority over othermemory accesses. Bus arbitration performing this function is well knownto one skilled in the art.

With further reference to FIG. 5, each of the memory interfaces 470 a–dincludes a respective memory controller 480, a respective write buffer482, and a respective cache memory unit 484. The memory controller 480performs the same functions as a conventional memory controller byproviding control, address and data signals to the memory devices 160 towhich it is coupled and receiving data signals from the memory device160 to which it is coupled. However, the nature of the signals sent andreceived by the memory controller 480 will correspond to the nature ofthe signals that the memory devices re adapted to send and receive. Thecache memory unit 484 includes the normal components of a cache memory,including a tag memory, a data memory, a comparator, and the like, as iswell known in the art. The memory devices used in the write buffer 482and the cache memory unit 484 may be either DRAM devices, static randomaccess memory (“SRAM”) devices, other types of memory devices, or acombination of all three. Furthermore, any or all of these memorydevices as well as the other components used in the cache memory unit484 may be either embedded or stand-alone devices.

The write buffer 482 in each memory interface 470 a–d is used to storewrite requests while a read request is being serviced. In such a system,the processor 104 can issue a write request to a system memory device440 a–d even if the memory device to which the write request is directedis busy servicing a prior write or read request. The write buffer 482preferably accumulates several write requests received from the switch460, which may be interspersed with read requests, and subsequentlyapplies them to each of the memory devices 160 in sequence without anyintervening read requests. By pipelining the write requests in thismanner, they can be more efficiently processed since delays inherent inread/write turnarounds are avoided. The ability to buffer write requeststo allow a read request to be serviced can also greatly reduce memoryread latency since read requests can be given first priority regardlessof their chronological order.

The use of the cache memory unit 484 in each memory interface 470 a–dallows the processor 104 to receive data responsive to a read commanddirected to a respective system memory device 160 without waiting forthe memory device 160 to provide such data in the event that the datawas recently read from or written to that memory device 160. The cachememory unit 484 thus reduces the read latency of the system memorydevices 440 a–d to maximize the memory bandwidth of the computer system.Similarly, the processor 104 can store write data in the cache memoryunit 484 and then perform other functions while the memory controller480 in the same memory interface 470 a–d transfers the write data fromthe cache memory unit 484 to the memory device 160 to which it iscoupled.

Further included in the memory hub local 150 may be a self-test module490 coupled to the switch 460 through a test bus 492. The self-testmodule 490 is further coupled to a maintenance bus 496, such as a SystemManagement Bus (SMBus) or a maintenance bus according to the Joint TestAction Group (JTAG) and IEEE 1149.1 standards. Both the SMBus and JTAGstandards are well known by those ordinarily skilled in the art.Generally, the maintenance bus 496 provides a user access to theself-test module 490 in order to set memory testing parameters andreceive test results. For example, the user can couple a separate PChost via the maintenance bus 496 to set the relative timing betweensignals that are applied to the memory devices 160. Similarly, dataindicative of the relative timing between signals that are received fromthe memory devices 160 can be coupled to the PC host via the maintenancebus 496.

Further included in the memory hub 150 may be a DMA engine 486 coupledto the switch 460 through a bus 488. The DMA engine 486 enables thememory hub 130 to move blocks of data from one location in one of thememory devices 160 to another location in the memory device withoutintervention from the processor 104. The bus 488 includes a plurality ofconventional bus lines and signal lines, such as address, control, databuses, and the like, for handling data transfers in the system memory.Conventional DMA operations well known by those ordinarily skilled inthe art can be implemented by the DMA engine 486.

From the foregoing it will be appreciated that, although specificembodiments of the invention have been described herein for purposes ofillustration, various modifications may be made without deviating fromthe spirit and scope of the invention. Accordingly, the invention is notlimited except as by the appended claims.

1. A method of operating a memory system having a memory hub controllercoupled to a plurality of memory hubs each of which includes at leastone receiver that is initialized before use, the method comprising:transmitting a downstream initialization complete signal from the memoryhub controller to one of the memory hubs when initialization of areceiver in the memory hub controller has been completed; determining ifa respective downstream initialization complete signal is being receivedby each of the memory hubs; determining if initialization of at leastone receiver in each of the memory hubs has been completed; if therespective downstream initialization complete signal is being receivedby each of the memory hubs except for a final memory hub furthestdownstream from the memory hub controller and initialization of at leastone receiver in the memory hub has been completed, transmitting adownstream initialization complete signal from the memory hub; if therespective downstream initialization complete signal is being receivedby the final memory hub and initialization of at least one receiver inthe final memory hub has been completed, transmitting an upstreaminitialization complete signal from the memory hub; if the respectiveupstream initialization complete signal is being received by each of thememory hubs determining if an upstream initialization complete signal isbeing received by the memory hub; if the respective upstreaminitialization complete signal is being received by each of the memoryhubs and initialization of all receivers in the memory hub has beencompleted, transmitting an upstream initialization complete signal fromthe memory hub; and if a respective upstream initialization completesignal is being received by the memory hub controller, transitioning thememory system to normal operation.
 2. The method of claim 1, furthercomprising switching each of the memory hubs to a normal operating moderesponsive to all of the memory hubs transmitting a respective upstreaminitialization complete signal.
 3. The method of claim 1 wherein each ofthe initialization complete signals comprises the transitioning of apredetermined series of binary values to a complement of the binaryvalues.
 4. The method of claim 3 wherein each of the predeterminedseries of binary values are used by the memory hub controller and thememory hubs initialize the receivers in the memory hub controller andthe memory hubs.
 5. The method of claim 3 wherein each of thepredetermined series of binary values and the complement of each of thepredetermined series of binary values are used by the memory hubcontroller and the memory hubs initialize the receivers in the memoryhub controller and the memory hubs.
 6. The method of claim 1, furthercomprising: transmitting from the memory hub controller and each of thememory hubs a respective initialization signal; and receiving at thememory hub controller and each of the memory hubs a respective one ofthe initialization signals.
 7. The method of claim 6 wherein each of theinitialization complete signals comprises a change in the nature of arespective one of the initialization signals.
 8. The method of claim 7wherein each of the initialization signals comprises a predeterminedseries of binary values.
 9. The method of claim 8 wherein the change inthe nature of each of the initialization signals comprises atransitioning of the predetermined series of binary values to acomplement of the predetermined series of binary values.
 10. The methodof claim 1 wherein a plurality of the memory hubs each comprise anupstream receiver, an upstream transmitter, a downstream receiver, anddownstream transmitter.
 11. In a memory system having at least onememory hub each having a downstream receiver, a downstream transmitter,an upstream transmitter and an upstream receiver, and a memory hubcontroller having a downstream transmitter and an upstream receiver, thememory hub controller having its downstream transmitter coupled to thedownstream receiver of a first of the memory hubs and its upstreamreceiver coupled to the upstream transmitter of the first of the memoryhubs, each of the memory hubs having its downstream receiver coupled tothe downstream transmitter of an upstream one of the memory modules, itsupstream transmitter coupled to the upstream receiver of the upstreamone of the memory modules, each of the receivers including aninitialization circuit for initializing the respective receiver, amethod of determining when all of the receivers have been initialized,comprising: in the memory hub controller: transmitting a downstreaminitialization complete signal from the downstream transmitter when theinitialization of the upstream receiver has been completed; in each ofthe memory hubs except for a memory hub furthest downstream from thememory hub controller: determining if a downstream initializationcomplete signal is being received by the downstream receiver;determining if initialization of at least one of the downstream receiverand the upstream receiver has been completed; and if the downstreaminitialization complete signal is being received by the downstreamreceiver and initialization of at least one of the downstream receiverand the upstream receiver has been completed, transmitting a downstreaminitialization complete signal from the downstream transmitter; in thememory hub furthest downstream from the memory hub controller:determining if a downstream initialization complete signal is beingreceived by the downstream receiver; determining if initialization of atleast the downstream receiver has been completed; and if the downstreaminitialization complete signal is being received by the downstreamreceiver and initialization of at least the downstream receiver has beencompleted, transmitting an upstream initialization complete signal fromthe upstream transmitter; in each of the memory hubs except for a memoryhub furthest downstream from the memory hub controller: determining ifan upstream initialization complete signal is being received by theupstream receiver; and if the upstream initialization complete signal isbeing received by the upstream receiver and initialization of both thedownstream receiver and the upstream receiver has been completed,transmitting an upstream initialization complete signal from theupstream transmitter; and in the memory hub controller: determining ifan upstream initialization complete signal is being received by theupstream receiver; and if the upstream initialization complete signal isbeing received by the upstream receiver, determining that all of thereceivers in the memory hub controllers and all of the memory hubs havebeen initialized.
 12. The method of claim 11 wherein the act ofdetermining if initialization of at least one of the downstream receiverand the upstream receiver has been completed comprises determining ifinitialization of the downstream receiver has been completed.
 13. Themethod of claim 11 wherein the act of determining if initialization ofat least one of the downstream receiver and the upstream receiver hasbeen completed comprises determining if initialization of both thedownstream receiver and the upstream receiver have been completed. 14.The method of claim 11, further comprising switching the memory hubcontroller to a normal operating mode responsive to the memory hubcontroller determining that all of the receivers in the memory hubcontrollers and all of the memory hubs have been initialized.
 15. Themethod of claim 11, further comprising switching each of the memory hubsto a normal operating mode responsive to the upstream transmitter in therespective memory hub transmitting a respective upstream initializationcomplete signal.
 16. The method of claim 11 wherein each of theinitialization complete signals comprises the transitioning of apredetermined series of binary values to a complement of the binaryvalues.
 17. The method of claim 16 wherein each of the predeterminedseries of binary values are used by the memory hub controller and thememory hubs initialize the receivers in the memory hub controller andthe memory hubs.
 18. The method of claim 16 wherein each of thepredetermined series of binary values and the complement of each of thepredetermined series of binary values are used by the memory hubcontroller and the memory hubs initialize the receivers in the memoryhub controller and the memory hubs.
 19. The method of claim 11, furthercomprising: transmitting from each of the transmitters an initializationsignal; and receiving at each of the receivers a respective one of theinitialization signals transmitted by a respective transmitter.
 20. Themethod of claim 19 wherein each of the initialization complete signalscomprises a change in the nature of a respective one of theinitialization signals.
 21. The method of claim 20 wherein each of theinitialization signals comprises a predetermined series of binaryvalues.
 22. The method of claim 21 wherein the change in the nature ofeach of the initialization signals comprises a transitioning of thepredetermined series of binary values to a complement of thepredetermined series of binary values.
 23. A method of operation of amemory hub controller including a transmitter and a receiver having aninitialization circuit for initializing the receiver, the methodcomprising: transmitting an initialization complete signal from thetransmitter when the initialization of the receiver has been completed;determining if the initialization complete signal is being received bythe receiver; and if the initialization complete signal is beingreceived by the receiver, transitioning the memory hub controller to anormal operating mode.
 24. The method of claim 23 wherein the receivedinitialization complete signal comprises the transitioning of apredetermined series of binary values to a complement of the binaryvalues.
 25. The method of claim 24, further comprising using thepredetermined series of binary values to initialize the receiver in thememory hub controller.
 26. The method of claim 25, further comprisingusing the complement of the predetermined series of binary values toinitialize the receiver in the memory hub controller.
 27. The method ofclaim 23, further comprising: transmitting from the transmitter arespective initialization signal; receiving at the receiver a respectiveinitialization signal; and using the initialization signal received bythe receiver to initialize the receiver.
 28. The method of claim 27wherein each of the initialization complete signals comprises a changein the nature of a respective one of the initialization signals.
 29. Themethod of claim 28 wherein each of the initialization signals comprisesa predetermined series of binary values.
 30. The method of claim 29wherein the change in the nature of each of the initialization signalscomprises transitioning the respective predetermined series of binaryvalues to a complement of the predetermined series of binary values. 31.A method of operation a memory hub including an upstream transmitter, adownstream transmitter, an upstream receiver, and a downstream receiver,the method comprising: determining if a downstream initializationcomplete signal is being received by the downstream receiver;determining if initialization of at least one of the downstream receiverand the upstream receiver has been completed; if the downstreaminitialization complete signal is being received by the downstreamreceiver and initialization of at least one of the downstream receiverand the upstream receiver has been completed, transmitting a downstreaminitialization complete signal from the downstream transmitter;determining if an upstream initialization complete signal is beingreceived by the upstream receiver; and if the upstream initializationcomplete signal is being received by the upstream receiver andinitialization of both the downstream receiver and the upstream receiverhas been completed, transmitting an upstream initialization completesignal from the upstream transmitter.
 32. The method of claim 31,further comprising transitioning the memory hub to a normal operatingmode responsive to the upstream receiver receiving the upstreaminitialization complete signal and initialization completion of both thedownstream receiver and the upstream receiver.
 33. The method of claim31 wherein each of the initialization complete signals comprises thetransitioning of a predetermined series of binary values to a complementof the binary values.
 34. The method of claim 33, further comprisingusing the predetermined series of binary values to initialize theupstream and downstream receivers in the memory hub.
 35. The method ofclaim 34, further comprising using the complement of the predeterminedseries of binary values to initialize the upstream and downstreamreceivers in the memory hub.
 36. The method of claim 31, furthercomprising: transmitting from the upstream transmitters an upstreaminitialization signal; transmitting from the downstream transmitter adownstream initialization signal; receiving at the upstream receiver anupstream initialization signal; receiving at the downstream receiver adownstream initialization signal; using the upstream initializationsignal to initialize the upstream receiver; and using the downstreaminitialization signal to initialize the downstream receiver.
 37. Themethod of claim 36 wherein each of the initialization complete signalscomprises a change in the nature of a respective one of theinitialization signals.
 38. The method of claim 37 wherein each of theinitialization signals comprises a predetermined series of binaryvalues.
 39. The method of claim 38 wherein the change in the nature ofeach of the initialization signals comprises transitioning therespective predetermined series of binary values to a complement of thepredetermined series of binary values.
 40. A memory system, comprising:a memory hub controller, comprising: an upstream receiver including aninitialization circuit for initializing the receiver, the upstreamreceiver being operable to determine if an upstream initializationcomplete signal is being received by the upstream receiver; and adownstream transmitter transmitting a downstream initialization completesignal when the initialization of the upstream receiver has beencompleted; a terminating memory module, comprising: a plurality ofmemory devices; and a memory hub coupled to the plurality of memorydevices, the memory hub comprising: a downstream receiver including aninitialization circuit for initializing the receiver, the downstreamreceiver being operable to determine if a downstream initializationcomplete signal is being received by the downstream receiver, thedownstream receiver further being operable to determine ifinitialization of the downstream receiver has been completed; and anupstream transmitter transmitting an upstream initialization completesignal from the upstream transmitter if the downstream initializationcomplete signal is being received by the downstream receiver andinitialization of at least the downstream receiver has been completed;and at least one intermediate memory module, each intermediate memorymodule comprising: a plurality of memory devices; and a memory hubcoupled to the plurality of memory devices, the memory hub comprising: adownstream receiver coupled to the downstream transmitter in either thememory hub controller or in the memory hub of another of theintermediate memory modules, the downstream receiver including aninitialization circuit for initializing the receiver, the downstreamreceiver being operable to determine if a downstream initializationcomplete signal is being received by the downstream receiver, thedownstream receiver further being operable to determine ifinitialization of at least one of the downstream receiver and theupstream receiver has been completed; a downstream transmitter coupledto the downstream receiver in the memory hub of either the terminatingmemory module or another of the intermediate memory modules, thedownstream transmitter being operable to transmit a downstreaminitialization complete signal if the downstream initialization completesignal is being received by the downstream receiver and initializationof at least one of the downstream receiver and the upstream receiver hasbeen completed; an upstream receiver coupled to the upstream transmitterin the memory hub of either the terminating memory module or another ofthe intermediate memory modules, the upstream receiver including aninitialization circuit for initializing the receiver, the upstreamreceiver being operable to determine if an upstream initializationcomplete signal is being received by the upstream receiver; and anupstream transmitter coupled to the upstream receiver in either thememory hub controller or in the memory hub of another of theintermediate memory modules, the upstream transmitter being operable totransmit an upstream initialization complete signal if an upstreaminitialization complete signal is being received by the upstreamreceiver and initialization of both the downstream receiver and theupstream receiver has been completed.
 41. The memory system of claim 40wherein the memory devices in each of the memory modules comprisedynamic random access memory devices.
 42. The memory system of claim 40wherein the downstream transmitter in the memory hub of each of theintermediate memory modules is operable to transmit the downstreaminitialization complete signal if the downstream initialization completesignal is being received by the downstream receiver and initializationof the downstream receiver but not necessarily the upstream receiver hasbeen completed.
 43. The memory system of claim 40 wherein the downstreamtransmitter in the memory hub of each of the intermediate memory modulesis operable to transmit the downstream initialization complete signal ifthe downstream initialization complete signal is being received by thedownstream receiver and initialization of both the downstream receiverand the upstream receiver has been completed.
 44. The memory system ofclaim 40 wherein the memory hub controller is further operable to switchthe memory hub controller to a normal operating mode responsive to thereceiver in the memory hub controller receiving the upstreaminitialization complete signal.
 45. The memory system of claim 40wherein each of the initialization complete signals comprises thetransitioning of a predetermined series of binary values to a complementof the binary values.
 46. The memory system of claim 45 wherein thereceivers in the memory hub controller and in each of the memory hubsare operable to use a respective one of the predetermined series ofbinary values to initialize the receivers.
 47. The memory system ofclaim 45 wherein the receivers in the memory hub controller and in eachof the memory hubs are operable to use a respective one of thepredetermined series of binary values and the complement of therespective one of the predetermined series of binary values toinitialize the receivers.
 48. The memory system of claim 40 wherein eachof the transmitters is further operable to transmit a respectiveinitialization signal, and each of the receivers is operable to receivea respective one of the initialization signals transmitted by arespective transmitter.
 49. The memory system of claim 48 wherein eachof the initialization complete signals comprises a change in the natureof a respective one of the initialization signals.
 50. The memory systemof claim 49 wherein each of the initialization signals comprises apredetermined series of binary values.
 51. The memory system of claim 50wherein the change in the nature of each of the initialization signalscomprises a transitioning of the predetermined series of binary valuesto a complement of the predetermined series of binary values.
 52. Amemory module, comprising: a plurality of memory devices; and a memoryhub coupled to the plurality of memory devices, the memory hubcomprising: a downstream receiver including an initialization circuitfor initializing the receiver, the downstream receiver being operable todetermine if a downstream initialization complete signal is beingreceived by the downstream receiver, the downstream receiver furtherbeing operable to determine if initialization of the downstream receiverhas been completed; a downstream transmitter coupled to the downstreamreceiver in the memory hub, the downstream transmitter being operable totransmit a downstream initialization complete signal if the downstreaminitialization complete signal is being received by the downstreamreceiver and initialization of the downstream receiver has beencompleted; an upstream receiver including an initialization circuit forinitializing the receiver, the upstream receiver being operable todetermine if an upstream initialization complete signal is beingreceived by the upstream receiver; and an upstream transmitter coupledto the upstream receiver, the upstream transmitter being operable totransmit an upstream initialization complete signal if an upstreaminitialization complete signal is being received by the upstreamreceiver and initialization of both the downstream receiver and theupstream receiver has been completed.
 53. The memory module of claim 52wherein the memory devices comprise dynamic random access memorydevices.
 54. The memory module of claim 52 wherein the downstreamtransmitter is operable to transmit the downstream initializationcomplete signal if the downstream initialization complete signal isbeing received by the downstream receiver and initialization of both thedownstream receiver and the upstream receiver has been completed. 55.The memory module of claim 52 wherein each of the initializationcomplete signals comprises the transitioning of a predetermined seriesof binary values to a complement of the binary values.
 56. The memorymodule of claim 55 wherein the receivers are operable to use arespective one of the predetermined series of binary values toinitialize the receivers.
 57. The memory module of claim 55 wherein thereceivers are operable to use a respective one of the predeterminedseries of binary values and the complement of the respective one of thepredetermined series of binary values to initialize the receivers. 58.The memory module of claim 52 wherein each of the transmitters isfurther operable to transmit a respective initialization signal, andeach of the receivers is operable to receive a respective one of theinitialization signals.
 59. The memory module of claim 58 wherein eachof the initialization complete signals comprises a change in the natureof a respective one of the initialization signals.
 60. The memory moduleof claim 59 wherein each of the initialization signals comprises apredetermined series of binary values.
 61. The memory module of claim 60wherein the change in the nature of each of the initialization signalscomprises a transitioning of the predetermined series of binary valuesto a complement of the predetermined series of binary values.
 62. Aprocessor-based system, comprising: a processor having a processor bus;a system controller coupled to the processor bus, the system controllerhaving a peripheral device port; at least one input device coupled tothe peripheral device port of the system controller; at least one outputdevice coupled to the peripheral device port of the system controller;at least one data storage device coupled to the peripheral device portof the system controller; and a first upstream data bus; a firstdownstream data bus; a memory hub controller coupled to the processorbus, the memory hub controller comprising: an upstream receiverincluding an initialization circuit for initializing the receiver, theupstream receiver being operable to determine if an upstreaminitialization complete signal is being received by the upstreamreceiver; and a downstream transmitter transmitting a downstreaminitialization complete signal when the initialization of the upstreamreceiver has been completed; a terminating memory module, comprising: aplurality of memory devices; and a memory hub coupled to the pluralityof memory devices, the memory hub comprising: a downstream receiverincluding an initialization circuit for initializing the receiver, thedownstream receiver being operable to determine if a downstreaminitialization complete signal is being received by the downstreamreceiver, the downstream receiver further being operable to determine ifinitialization of the downstream receiver has been completed; and anupstream transmitter transmitting an upstream initialization completesignal from the upstream transmitter if the downstream initializationcomplete signal is being received by the downstream receiver andinitialization of at least the downstream receiver has been completed;and at least one intermediate memory module, each intermediate memorymodule comprising: a plurality of memory devices; and a memory hubcoupled to the plurality of memory devices, the memory hub comprising: adownstream receiver coupled to the downstream transmitter in either thememory hub controller or in the memory hub of another of theintermediate memory modules, the downstream receiver including aninitialization circuit for initializing the receiver, the downstreamreceiver being operable to determine if a downstream initializationcomplete signal is being received by the downstream receiver, thedownstream receiver further being operable to determine ifinitialization of at least one of the downstream receiver and theupstream receiver has been completed; a downstream transmitter coupledto the downstream receiver in the memory hub of either the terminatingmemory module or another of the intermediate memory modules, thedownstream transmitter being operable to transmit a downstreaminitialization complete signal if the downstream initialization completesignal is being received by the downstream receiver and initializationof at least one of the downstream receiver and the upstream receiver hasbeen completed; an upstream receiver coupled to the upstream transmitterin the memory hub of either the terminating memory module or another ofthe intermediate memory modules, the upstream receiver including aninitialization circuit for initializing the receiver, the upstreamreceiver being operable to determine if an upstream initializationcomplete signal is being received by the upstream receiver; and anupstream transmitter coupled to the upstream receiver in either thememory hub controller or in the memory hub of another of theintermediate memory modules, the upstream transmitter being operable totransmit an upstream initialization complete signal if an upstreaminitialization complete signal is being received by the upstreamreceiver and initialization of both the downstream receiver and theupstream receiver has been completed.
 63. The processor-based system ofclaim 62 wherein the memory devices in each of the memory modulescomprise dynamic random access memory devices.
 64. The memory system ofclaim 62 wherein the downstream transmitter in the memory hub of each ofthe intermediate memory modules is operable to transmit the downstreaminitialization complete signal if the downstream initialization completesignal is being received by the downstream receiver and initializationof the downstream receiver but not necessarily the upstream receiver hasbeen completed.
 65. The memory system of claim 62 wherein the downstreamtransmitter in the memory hub of each of the intermediate memory modulesis operable to transmit the downstream initialization complete signal ifthe downstream initialization complete signal is being received by thedownstream receiver and initialization of both the downstream receiverand the upstream receiver has been completed.
 66. The memory system ofclaim 62 wherein the memory hub controller is further operable to switchthe memory hub controller to a normal operating mode responsive to thereceiver in the memory hub controller receiving the upstreaminitialization complete signal.
 67. The memory system of claim 62wherein each of the initialization complete signals comprises thetransitioning of a predetermined series of binary values to a complementof the binary values.
 68. The memory system of claim 67 wherein thereceivers in the memory hub controller and in each of the memory hubsare operable to use a respective one of the predetermined series ofbinary values to initialize the receivers.
 69. The memory system ofclaim 67 wherein the receivers in the memory hub controller and in eachof the memory hubs are operable to use a respective one of thepredetermined series of binary values and the complement of therespective one of the predetermined series of binary values toinitialize the receivers.
 70. The memory system of claim 62 wherein eachof the transmitters is further operable to transmit a respectiveinitialization signal, and each of the receivers is operable to receivea respective one of the initialization signals transmitted by arespective transmitter.
 71. The memory system of claim 70 wherein eachof the initialization complete signals comprises a change in the natureof a respective one of the initialization signals.
 72. The memory systemof claim 71 wherein each of the initialization signals comprises apredetermined series of binary values.
 73. The memory system of claim 72wherein the change in the nature of each of the initialization signalscomprises a transitioning of the predetermined series of binary valuesto a complement of the predetermined series of binary values.